Facial Expression Recognition for Mental Health Monitoring

Abstract

Anxiety, depression, and stress are examples of mental health issues that have become major global health concerns, impacting millions of people globally. Early identification and ongoing monitoring of mental health issues are now essential due to the rising need for prompt and efficient mental health care. Conventional techniques for evaluating mental health, such self-report questionnaires and clinical interviews, frequently depend on subjective information and are therefore open to bias or inconsistent results. More objective, timely, and non-invasive techniques for monitoring mental health are therefore required. As per WHO Mental health issues encompass a range of mental states and illnesses, including psychosocial impairments and mental disorders, that are linked to considerable suffering, impaired functioning, or self-harm risk. 9 million individuals worldwide suffered from a mental illness in 2019, with anxiety and depression being the most prevalent. Facial expression recognition (FER) has drawn a lot of interest in this area as a potentially useful method for tracking psychological and emotional states. Facial expressions may be useful indications of an individual\'s emotional state because they are intimately related to it. An extensive analysis of contactless sensing techniques for tracking mental health is provided in this article. It looks at published studies that employ contactless sensing techniques to forecast mental health conditions.

Introduction

I. INTRODUCTION

Emotion recognition plays a crucial role in enhancing human-machine interaction. The inherent complexity of emotions makes their detection and interpretation particularly challenging. However, advancements in deep learning, particularly through neural networks, have significantly improved the accuracy of emotion recognition. Recent developments in deep learning have incorporated various types of human behavior inputs, including auditory and visual data, facial expressions, body gestures, as well as EEG and other brainwave signals [8].

Facial expressions are not only a natural way to display human emotions but also serve as a crucial method of non-verbal communication. Mental health, which is vital to overall well-being, has become a significant global concern. The rising incidence of mental health conditions like anxiety, depression, and stress highlights the urgent need for better monitoring and intervention strategies [14]. Traditional mental health assessments, which rely on self-reports and clinical interviews, often lack objectivity, sensitivity, and accessibility [6]. As a result, there is an increasing demand for innovative approaches that can offer more accurate, timely, and non-invasive ways to assess mental health.

We have aimed to examine key related work, their techniques, the effectiveness of these methods, and the potential for enhancing the outcomes.

The Promise of Facial Expression Recognition (FER)

Facial expression recognition (FER), a branch of computer vision, has emerged as a potential solution to some of these challenges [4]. By analyzing facial expressions, FER systems can provide valuable insights into a person's emotional state, enabling a continuous and objective mental health assessment. The human face can express a wide range of emotions through subtle shifts in muscle movements and facial features, and FER technology uses these cues to extract meaningful data on emotional well-being.

Deep convolutional neural networks (CNNs) have demonstrated superior performance over traditional methods in various visual recognition tasks, including facial expression recognition (FER) [18]. However, despite attempts to enhance the accuracy of FER systems using CNNs, existing approaches may still fall short for practical use. This study provides a comprehensive review of FER systems employing CNNs, highlighting both their advantages and limitations

A. FER presents several benefits compared to traditional mental health assessment techniques:

1. Objectivity: FER provides an unbiased evaluation of emotional states, minimizing the subjectivity and intentional distortions often seen in self-reports [7].
2. Real-time monitoring: FER systems can track facial expressions in real-time, allowing for immediate detection of emotional changes, which is particularly useful for those experiencing mood swings or at risk of self-harm.
3. Non-invasive: FER is a non-contact method, making it a more convenient and accessible option for mental health monitoring.
4. Accessibility: FER technology can be incorporated into everyday devices such as smartphones or wearables, broadening its accessibility to diverse populations
5. Early detection: By detecting subtle facial changes, FER may help identify early signs of mental health issues, facilitating timely interventions and possibly preventing the escalation of more severe conditions.

???????B. Challenges and Limitations of FER

1. Accuracy: Interpreting facial expressions, particularly subtle or unclear ones, can be difficult, and factors like lighting, obstructions, and individual variability may affect the performance of FER algorithms.
2. Cultural Differences: Facial expressions can differ between cultures, making it hard to develop FER models that apply universally across different groups [12].
3. Individual Variability: Even within the same culture, people express emotions in unique ways, complicating efforts to create standardized interpretations of facial expressions.
4. Privacy issues: Collecting and analyzing facial data raises privacy concerns, especially when used for mental health tracking. It is critical to ensure FER technology is used ethically and responsibly [2].
5. Ethical concerns: FER technology could potentially be misused for purposes such as discrimination or unwarranted surveillance, underscoring the need for clear guidelines and regulations to prevent such abuses [13].

Despite these obstacles, FER remains a promising research area with the potential to significantly advance mental health monitoring tools. As researchers continue to address its limitations and ethical challenges, there is potential to refine FER technology and explore its use in various mental health contexts. This review paper will examine the latest FER techniques and their applications in mental health monitoring. It will also explore the challenges and limitations associated with FER, while discussing the potential benefits and ethical implications of using this technology in clinical settings [17]. By evaluating the current state of FER research and its mental health applications, this paper aims to contribute to the development of innovative strategies for improving mental health outcomes.

II. BODY

Facial emotion recognition plays a crucial role in classifying mental health states, leading to the development of numerous methods for emotion recognition. Computer vision is frequently linked to facial expression recognition, as it aids in analyzing images and classifying them [20]. This topic is elaborated upon based on various studies related to facial expression recognition and computer vision. Facial Expression Recognition (FER) is a focused area within computer vision that aims to detect and categorize facial expressions from images or video feeds. FER’s main purpose is to understand human emotions by examining slight changes in facial muscles and movements [1]. This process involves pinpointing crucial facial landmarks like the eyes, eyebrows, mouth, and nose, and analyzing their positions and shifts to infer emotional states [11]. Common emotions identified by FER systems include happiness, sadness, anger, surprise, fear, and disgust, typically grounded in Ekman’s basic emotions theory.

A standard pipeline for Facial Expression Recognition (FER) begins with face detection, where algorithms separate the face from the background. Next, feature extraction targets key facial details relevant to emotions, such as the contours and positions of facial features [3]. Finally, a classification model, typically driven by machine learning, analyzes these extracted features to identify the expression. While early FER systems used handcrafted features like Gabor filters and Local Binary Patterns for this task, modern FER leverages deep learning, especially Convolutional Neural Networks (CNNs), to automate feature extraction and enhance accuracy in recognizing expressions. The field of computer vision is instrumental in this process, offering advanced techniques for analyzing and interpreting facial expressions [15]. Recent innovations in this domain continue to advance the effectiveness and application of emotion recognition technologies, supporting more nuanced and comprehensive mental health assessments.

1) Face Detection: The system takes a human face as input, captured through a webcam. The captured image undergoes enhancement, where tone mapping is applied to improve contrast in low-contrast images.

1. Binarization: All RGB and grayscale images are converted into binary form. Figure 2 illustrates the binarization process of a human face. The preprocessed binary image is then passed to the face detection stage.
2. Segmentation: A bounding box is created around each facial feature that contributes to emotional expression, using face coordinates. These segmented boxes are then analyzed to determine emotions. Figure 3 shows the complete facial segmentation process.

2) Feature Extraction

The facial image from the detection phase serves as input for feature extraction. To reduce complexity and enable real-time performance, only the eyes and mouth are considered, as these features effectively convey emotions. A corner point detection algorithm is applied to identify key points in these regions.

1. 1. 1. Eye Extraction: Eyes exhibit strong vertical edges due to the iris and eye white. The Sobel mask is applied to the image, and the Y-coordinate of the eyes is determined through the horizontal projection of these vertical edges.
      2. Eyebrow Extraction: Two rectangular areas above the eyes are selected as the eyebrow regions. Edge images of these areas are refined using the Sobel method, which detects more edges than the Roberts method. The obtained edge images are then dilated, and holes are filled to further refine the eyebrow regions.
      3. Mouth Extraction: The top, bottom, left, and rightmost points of the mouth are identified, and the centroid of the mouth is calculated. Figure 5 illustrates the mouth extraction process.

3) Facial Emotion Recognition: An emotion matrix is created, with rows and columns representing the intensity of various emotions. This matrix establishes a range for detecting different combinations of emotional states.

4) Mental State detection: Recognized emotions are stored and monitored over time. Based on the persistence and recurrence of these emotions, the individual's mental state is assessed. A psychological test is then administered to evaluate the severity of any potential clinical condition.

5) Evaluation of Psychometric: Psychometrics involves designing and interpreting tests that measure psychological traits such as aptitude, personality, memory, happiness, and intelligence. These tests are commonly used in mental health, education, and employment settings. In this project, IQ and personality traits are measured using Hans Eysenck’s personality test, which assesses three dimensions: Psychoticism, Extraversion, and Neuroticism (PEN). Personality disorders such as depression, anxiety, and dissociation can be detected early, allowing for preventive measures..

6) Classification of Mental Illness: Based on the results of psychometric tests and the detected mental state, the likelihood and nature of mental illness or criminal behavior are classified. Conditions may be identified as either self-destructive or harmful to society.

III. EXISTING TECHNOLOGIES

Facial Expression Recognition (FER) has advanced considerably over time. Early research concentrated on rule-based and statistical techniques for interpreting facial features, but these initial methods had limited success, frequently facing challenges with lighting, orientation, and occlusions in real-world conditions [10]. The adoption of machine learning marked a substantial improvement, with models like Support Vector Machines (SVMs) and Hidden Markov Models (HMMs) enhancing the classification of expressions from extracted features. Despite these advancements, these models still depended on manual feature engineering, which was labor-intensive and prone to data variability.

The rise of deep learning, especially Convolutional Neural Networks (CNNs), marked a significant breakthrough in Facial Expression Recognition (FER). CNNs enabled end-to-end learning, automating feature extraction and greatly improving expression recognition accuracy. Advanced architectures such as VGG, ResNet, and Inception have further expanded these capabilities, allowing FER models to handle complex variations in expressions and environmental conditions [5]. Recently, transformer-based models, including Vision Transformers (ViTs), have also shown promise in FER, offering improved contextual understanding and attention mechanisms that enhance recognition accuracy.

Facial expression recognition (FER) technologies have gained prominence in mental health monitoring. By analyzing facial cues, these technologies offer real-time insights into an individual’s mental state, making them crucial for the early detection and ongoing evaluation of conditions like anxiety, depression, and stress [10].

The primary models employed in FER today are

1. Convolutional Neural Networks (CNNs): A deep learning algorithm highly effective in image recognition tasks, CNNs are commonly employed in FER systems to automatically detect and classify facial expressions by analyzing pixel data from images or video.
2. Support Vector Machines (SVMs): A supervised machine learning technique used in FER for classifying emotions. SVMs work by identifying the optimal hyperplane that separates different emotion categories based on facial features.
3. Facial Action Coding System (FACS)): A manual system that categorizes facial movements into Action Units (AUs). Many FER systems use FACS to map these movements to specific emotions, offering a standardized way to detect emotional states.
4. 3D Facial Recognition: This technology analyzes the three-dimensional structure of facial features, providing more precise information about facial expressions compared to 2D methods, and is particularly useful for identifying subtle emotional cues [16].
5. Optical Flow Analysis: This technique tracks the movement of facial features over time, essential for detecting real-time changes in expressions. It is used in FER systems to continuously monitor emotional variations.

A. Psychological Basis of Facial Expressions in Mental Health:

Facial expressions are key indicators of emotional states and are essential for nonverbal communication. Psychologically, certain expressions are associated with specific emotions, offering valuable insights into an individual’s mental health [18]. For example, individuals with depression often show reduced smiling and limited eye contact, signaling a decrease in positive emotions. Likewise, people with anxiety may display expressions of fear, nervousness, or tension, often reflected in widened eyes or a tightened jaw.

Research indicates that specific mental health conditions are linked to distinctive facial expressions. For instance, individuals with bipolar disorder may exhibit shifting expressions that reflect their alternating manic and depressive states. People with PTSD may display expressions of distress or hypervigilance, indicating heightened alertness [19]. By examining these subtle cues, facial expression recognition (FER) can assist in monitoring and assessing mental health, offering a non-invasive approach for early detection and intervention.

Conclusion

Facial expression recognition (FER) has emerged as a valuable tool for monitoring mental health, providing a non-invasive and objective method for evaluating emotional states. Although FER technology has advanced significantly, there are still several challenges to overcome to ensure its broad adoption and effectiveness in clinical environments [20]. A. Key areas to focus on for the future of FER in mental health monitoring include: 1) Enhancing accuracy: Ongoing research is required to develop more precise FER algorithms that can effectively manage a wide range of facial expressions, varying lighting conditions, and individual differences. 2) Cultural adaptability: It is crucial to conduct cross-cultural research to ensure FER models are effective across different populations and accurately interpret facial expressions in diverse cultural contexts. 3) Privacy and ethical considerations: Implementing stringent guidelines and regulations is essential to protect individual privacy and prevent potential misuse of FER technology. 4) Integration with other assessment tools: FER should be combined with other mental health evaluation methods to offer a more comprehensive and accurate understanding of emotional states. 5) Exploring new uses: Future research should investigate additional applications of FER in mental health contexts, such as telepsychiatry, workplace wellness, and educational settings. In summary, FER holds great potential for advancing mental health monitoring and enhancing early detection and treatment of mental health issues. By addressing current challenges and limitations, researchers can develop more effective and accessible tools for mental health assessment. As FER technology progresses, it could transform how mental health is evaluated and managed, leading to better outcomes for individuals globally.

References

[1] Liu, Nan, Haihong Liu, and Haining Liu. \"Mental health diagnosis of college students based on facial recognition and neural network.\" Journal of Intelligent & Fuzzy Systems 40.4 (2021): 7061-7072. [2] Wang, Zixiang. \"Campus intelligence mental health searching system based on face recognition technology.\" Journal of Electronic Research and Application 4.3 (2020). [3] Veena, S., & Aravindhar, J. (2021). Mental Health Monitoring System Using Facial Recognition, PEN Test and IQ Test. [4] Raut, Mayank, Pranav Sawant, Anish Bandal, Suryansh Desai, and Yogesh Gaikwad. \"Depression Detection System Using Facial Recognition Usign Python.\" [5] Oburi, Naveen Kumar, et al. \"Early Identification of Mental Health Disorder Employing Machine Learning-based Secure Edge Analytics: A Real-time Monitoring System.\" Secure Edge Computing. CRC Press, 2021. 117-136. [6] Li, Chong, et al. \"An intelligent mental health identification method for college students: A mixed-method study.\" International Journal of Environmental Research and Public Health 19.22 (2022): 14976. [7] Abd Rahman, Rohizah, et al. \"Application of machine learning methods in mental health detection: a systematic review.\" Ieee Access 8 (2020): 183952-183964. [8] Singh, Sonali, and Navita Srivastava. \"EMOTION RECOGNITION FOR MENTAL HEALTH PREDICTION USING AI TECHNIQUES: AN OVERVIEW.\" International Journal of Advanced Research in Computer Science 14.3 (2023). [9] Jain, Manav Pradeep, Soumyaprakash Sribash Dasmohapatra, and Stevina Correia. \"Mental health state detection using open cv and sentimental analysis.\" 2020 3rd International Conference on Intelligent Sustainable Systems (ICISS). IEEE, 2020. [10] Peng, Xianglan. \"Research on Emotion Recognition Based on Deep Learning for Mental Health.\" Informatica 45.1 (2021). [11] Ming, Foo Jia, et al. \"Facial Emotion Recognition System for Mental Stress Detection among University Students.\" 2023 3rd International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME). IEEE, 2023. [12] Yang, Minqiang, et al. \"Undisturbed mental state assessment in the 5G era: a case study of depression detection based on facial expressions.\" IEEE Wireless Communications 28.3 (2021): 46-53. [13] Gilanie, Ghulam, et al. \"A Robust Method of Bipolar Mental Illness Detection from Facial Micro Expressions Using Machine Learning Methods.\" Intelligent Automation & Soft Computing 39.1 (2024). [14] Aina, Joseph Oluwagbogo. Mental Disorder Detection System Through Emotion Recognition. MS thesis. Morgan State University, 2023. [15] Saha, Tulika, et al. \"Mental health disorder identification from motivational conversations.\" IEEE Transactions on Computational Social Systems 10.3 (2022): 1130-1139. [16] Shanthi, N., et al. \"An integrated approach for mental health assessment using emotion analysis and scales.\" Healthcare Technology Letters (2022). [17] Wolf, Karsten. \"Measuring facial expression of emotion.\" Dialogues in clinical neuroscience 17.4 (2015): 457-462. [18] Nouman, M., Khoo, S. Y., Mahmud, M. P., & Kouzani, A. Z. (2021). Recent advances in contactless sensing technologies for mental health monitoring. IEEE Internet of Things Journal, 9(1), 274-297. [19] Tufail, Hina, et al. \"Depression Detection with Convolutional Neural Networks: A Step Towards Improved Mental Health Care.\" Procedia Computer Science 224 (2023): 544-549. [20] Qiao, Jiang. \"A systematic review of machine learning approaches for mental disorder prediction on social media.\" 2020 International Conference on Computing and Data Science (CDS). IEEE, 2020.

Copyright

Copyright © 2024 Inakshi Garg, Vaibhav Mishra, Arpit Singh. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.